Display control system and method, display control apparatus and method, and program

ABSTRACT

A display control system for controlling display of images on a plurality of display units, the plurality of display units being arranged in an array, includes a determining unit determining whether or not information of interest is being displayed; a first recording control unit controlling recording of a first image displayed on the display units when the determining unit determines that the information of interest is being displayed; a second recording control unit controlling recording of a second image displayed on the display units when a user performs a recording operation; and a display control unit controlling display of the first image or the second image on the display units.

CROSS REFERENCES TO RELATED APPLICATIONS

The present invention contains subject matter related to Japanese PatentApplication JP 2006-250428 filed in the Japanese Patent Office on Sep.15, 2006, the entire contents of which are incorporated herein byreference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to display control systems and methods,display control apparatuses and methods, and programs. Morespecifically, the present invention relates to a display control systemand method, a display control apparatus and method, and a program forallowing a user to easily record information of interest displayed onany of a plurality of displays and to display the recorded information.

2. Description of the Related Art

Recently, multi-display technology has become available. Themulti-display technology allows different images to be displayed onmultiple display devices of a multi-display apparatus arrangedvertically or horizontally in the same orientation. Further, suchmultiple display devices can be used in combination to display a singleimage in an enlarged scale, or a divided image can be displayed with adesired size (see Japanese Unexamined Patent Application Publication No.2004-289866).

Such a multi-display apparatus provides a “zapping” function once usersmove their line of sight while viewing programs. Users can browsethrough many programs by moving their line of sight, and can thereforeeasily find information of interest.

Japanese Unexamined Patent Application Publication No. 2004-213486discloses an image processing apparatus for controlling a process fordetecting a user's line of sight and selecting an image corresponding tothe detected line of sight from among a plurality of images displayed ondisplay means.

In general, display devices have a “memo” function. Users press a memobutton provided on a remote controller or the like when information ofinterest is displayed on the display devices, thereby storing theinformation in a memory. In particular, display devices further having azoom display function allow any small and illegible part of an image tobe stored using the memo function so that the stored part can beenlarged and displayed later using the zoom display function. Therefore,the memo function may be highly demanded.

SUMMARY OF THE INVENTION

However, multi-display apparatuses have a problem. As described above,the zapping function is provided once users move their line of sight.Therefore, as shown in FIG. 1, users may find information of interest atthe moment when they move the line of sight.

FIG. 1 shows an example in which an image is displayed on displaydevices constituting a multi-display apparatus. The multi-displayapparatus shown in FIG. 1 includes nine display devices 1-1 to 1-9, anddisplay screens of the display devices 1-1 to 1-9 are arranged in anarray. In the example shown in FIG. 1, for the convenience ofillustration, only the display device 1-3 displays an image; however,the other display devices 1-1, 1-2, and 1-4 to 1-9 also display imagesof different programs.

For example, a user A moves his/her line of sight from the imagedisplayed on the display device 1-2 to the image displayed on thedisplay device 1-3, and finds information of interest in the imagedisplayed on the display device 1-3. In the example shown in FIG. 1, apitcher during a game of baseball appears on the display device 1-3, andthe user A is interested in the pitcher's grip on the ball.

The user A desires to operate a remote controller 2 to use the memofunction. However, due to the time-consuming operation of the remotecontroller 2 or the delayed reaction of the user A, the user A may notbe able to use the memo function at the desired time and may fail tocatch the desired information. For example, in the multi-displayapparatus, the memo function is not available until the user A hasselected which display device displaying an image of which the user Adesires to make a memo, or the user A may hesitate to press the memobutton on the remote controller 2.

To address this problem, it is conceivable to constantly record imagesof the last several frames in a hard disk or the like. However, it istime-consuming to operate the hard disk to search for desiredinformation, and a large number of images are constantly recorded,resulting in an increase in the processing load. Further, the cost of ahard disk used for video recording is greater than the cost of a memoryused for the memo function.

It is therefore desirable to allow a user to easily record informationof interest with no failure when the user finds the information on amulti-display apparatus having a plurality of display devices.

According to a first embodiment of the present invention, there isprovided a display control system for controlling display of images on aplurality of display means, the plurality of display means beingarranged in an array. The display control system includes determiningmeans for determining whether or not information of interest is beingdisplayed; first recording control means for controlling recording of afirst image displayed on the display means when the determining meansdetermines that the information of interest is being displayed; secondrecording control means for controlling recording of a second imagedisplayed on the display means when a user performs a recordingoperation; and display control means for controlling display of thefirst image or the second image on the display means.

The display control system can further include a plurality of displaycontrol apparatuses each associated with each of the plurality ofdisplay means and controlling display of an image on the display means.Each of the plurality of display control apparatuses can include thedetermining means, the first recording control means, the secondrecording control means, and the display control means.

The display control system can further include the plurality of displaymeans.

The display control system can further include a line-of-sight detectionapparatus that detects a line of sight of the user, and the determiningmeans of each of the plurality of display control apparatuses candetermine whether or not the information of interest is being displayedon the display means on the basis of a detection result from theline-of-sight detection apparatus.

The first recording control means of each of the plurality of displaycontrol apparatuses can control the recording of the first imagedisplayed on the display means when the determining means determinesthat the information of interest is being displayed on the display meanson the basis of the detection result from the line-of-sight detectionapparatus.

The display control system can further include a remote controller thatcontrols the plurality of display control apparatuses, and the remotecontroller can include speech detecting means for detecting specificspeech of the user. The determining means of each of the plurality ofdisplay control apparatuses can determine whether or not the informationof interest is being displayed on one of the plurality of display meanson the basis of a detection result from the speech detecting means.

The first recording control means of each of the plurality of displaycontrol apparatuses can control the recording of the first imagedisplayed on the display means when the determining means determinesthat the information of interest is being displayed on one of theplurality of display means on the basis of the detection result from thespeech detecting means.

The display control system can further include a remote controller thatcontrols the plurality of display control apparatuses, and the remotecontroller can include acceleration detecting means for detecting anacceleration of the remote controller that is operated by the user. Thedetermining means of each of the plurality of display controlapparatuses can determine whether or not the information of interest isbeing displayed on one of the plurality of display means on the basis ofa detection result from the acceleration detecting means.

The first recording control means of each of the plurality of displaycontrol apparatuses can control the recording of the first imagedisplayed on the display means when the determining means determinesthat the information of interest is being displayed on one of theplurality of display means on the basis of the detection result from theacceleration detecting means.

The first recording control means of each of the plurality of displaycontrol apparatuses can control to start recording of a moving image,starting from the first image displayed on the display means, when thedetermining means determines that the information of interest is beingdisplayed. The second recording control means can control to stop therecording of the moving image at the second image displayed on thedisplay means when the user performs the recording operation. Thedisplay control means can control display of the moving image startingwith the first image displayed on the display means and ending with thesecond image displayed on the display means.

According to the embodiment of the present invention, there is provideda display control method for a display control system for controllingdisplay of images on a plurality of display means, the plurality ofdisplay means being arranged in an array. The display control methodincludes the steps of determining whether or not information of interestis being displayed; controlling recording of a first image displayed onthe display means when it is determined that the information of interestis being displayed; controlling recording of a second image displayed onthe display means when a user performs a recording operation; andcontrolling display of the first image or the second image on thedisplay means.

According to a second embodiment of the present invention, there isprovided a display control apparatus in a display control systemincluding a plurality of display control apparatuses, for controllingdisplay of an image on display means associated with the display controlapparatus from among a plurality of display means arranged in an array.The display control apparatus includes determining means for determiningwhether or not information of interest is being displayed; firstrecording control means for controlling recording of a first imagedisplayed on the display means when the determining means determinesthat the information of interest being displayed; second recordingcontrol means for controlling recording of a second image displayed onthe display means when a user performs a recording operation; anddisplay control means for controlling display of the first image or thesecond image on the display means.

According to the second embodiment of the present invention, there isprovided a display control method for a display control apparatus in adisplay control system including a plurality of display controlapparatuses, for controlling display of an image on display meansassociated with the display control apparatus from among a plurality ofdisplay means arranged in an array. The display control method includesthe steps of determining whether or not information of interest is beingdisplayed; controlling recording of a first image displayed on thedisplay means associated with the display control apparatus when it isdetermined that the information of interest is being displayed;controlling recording of a second image displayed on the display meansassociated with the display control apparatus when a user performs arecording operation; and controlling display of the first image or thesecond image on the display means associated with the display controlapparatus.

According to the second embodiment of the present invention, there isprovided a program for allowing a display control apparatus in a displaycontrol system including a plurality of display control apparatuses toexecute a process for controlling display of an image on display meansassociated with the display control apparatus from among a plurality ofdisplay means arranged in an array. The program includes the steps ofdetermining whether or not information of interest is being displayed;controlling recording of a first image displayed on the display meansassociated with the display control apparatus when it is determined thatthe information of interest is being displayed; controlling recording ofa second image displayed on the display means associated with thedisplay control apparatus when a user performs a recording operation;and controlling display of the first image or the second image on thedisplay means associated with the display control apparatus.

According to the first embodiment of the present invention, in a displaycontrol system for controlling display of images on a plurality ofdisplay means, the plurality of display means being arranged in anarray, it is determined whether or not information of interest is beingdisplayed, and the recording of a first image displayed on the displaymeans is controlled when it is determined that the information ofinterest is being displayed. In accordance with a recording operationperformed by a user, a second image displayed on the display means iscontrolled. The display of the first or second image on the displaymeans is controlled.

According to the second embodiment of the present invention, it isdetermined whether or not information of interest is being displayed,and the recording of a first image displayed on the display means iscontrolled when it is determined that the information of interest isbeing displayed. In accordance with a recording operation performed by auser, the recording of a second image displayed on the display means iscontrolled. The display of the first or second image on the displaymeans is controlled.

According to the first embodiment of the present invention, therefore,even in an environment where a plurality of display devices arearranged, a user can easily record information of interest with nofailure when the user finds the information being displayed on any ofthe display devices.

According to the second embodiment of the present invention, therefore,even in an environment where a plurality of display devices arearranged, a user can avoid failure to catch information of interest witha small memory capacity when the user finds the information beingdisplayed on any of the display devices.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram showing a memo function of a multi-display apparatusof the related art;

FIG. 2 is a block diagram showing an example of the structure of amulti-display system according to a first embodiment of the presentinvention;

FIG. 3 is an external view showing an example of the structure of aremote controller shown in FIG. 2;

FIG. 4 is a block diagram showing an example of the structure of adisplay device shown in FIG. 2;

FIG. 5 is a block diagram showing an example of the electrical structureof the remote controller shown in FIG. 2;

FIG. 6 is a diagram showing a process for recording an image usingline-of-sight information;

FIG. 7 is a diagram showing the process for recording an image usingline-of-sight information;

FIG. 8 is a flowchart showing a process of the display device shown inFIG. 2;

FIG. 9 is a flowchart showing a memory control process in step S13 shownin FIG. 8;

FIG. 10 is a flowchart showing a process in accordance with a command instep S19 shown in FIG. 8;

FIG. 11 is a flowchart showing a process of the remote controller shownin FIG. 2;

FIG. 12 is an external view showing another example of the structure ofthe remote controller shown in FIG. 2;

FIG. 13 is a diagram showing a process for recording a moving imageusing line-of-sight information;

FIG. 14 is a flowchart showing a process of the display device shown inFIG. 2 in the case shown in FIG. 13;

FIG. 15 is a flowchart showing a memory control process in step S73shown in FIG. 14;

FIG. 16 is a flowchart showing a memory write process in step S74 shownin FIG. 14;

FIG. 17 is a flowchart showing a process in accordance with a command instep S80 shown in FIG. 14;

FIG. 18 is an external view showing another example of the structure ofthe remote controller shown in FIG. 2 in the case shown in FIG. 13;

FIG. 19 is an external view showing another example of the structure ofthe remote controller shown in FIG. 2 in the case shown in FIG. 13;

FIG. 20 is a block diagram showing an example of the structure of amulti-display system according to a second embodiment of the presentinvention;

FIG. 21 is a block diagram showing an example of the structure of adisplay device shown in FIG. 20;

FIG. 22 is a block diagram showing an example of the electricalstructure of a remote controller shown in FIG. 20;

FIG. 23 is a diagram showing a process for recording an image using anevent-determination result flag;

FIG. 24 is a diagram showing the process for recording an image using anevent-determination result flag;

FIG. 25 is a flowchart showing a process of the remote controller shownin FIG. 20;

FIG. 26 is a flowchart showing an event-detection flag output process instep S152 shown in FIG. 25;

FIG. 27 is a flowchart showing a process of the display device shown inFIG. 20;

FIG. 28 is a flowchart showing a memory control process in step S203shown in FIG. 27;

FIG. 29 is a block diagram showing another example of the display deviceshown in FIG. 20;

FIG. 30 is a block diagram showing an example of the structure of anevent determination unit shown in FIG. 29;

FIG. 31 is a flowchart showing another example of the event-detectionflag output process in step S152 shown in FIG. 25; and

FIG. 32 is a block diagram showing an example of the structure of apersonal computer according to an embodiment of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Before describing an embodiment of the present invention, thecorrespondence between the features of the claims and the specificelements disclosed in an embodiment of the present invention isdiscussed below. This description is intended to assure that embodimentssupporting the claimed invention are described in this specification.Thus, even if an element in the following embodiments is not describedherein as relating to a certain feature of the present invention, thatdoes not necessarily mean that the element does not relate to thatfeature of the claims. Conversely, even if an element is describedherein as relating to a certain feature of the claims, that does notnecessarily mean that the element does not relate to other features ofthe claims.

According to a first embodiment of the present invention, there isprovided a display control system (e.g., a multi-display system shown inFIG. 2) for controlling display of images on a plurality of displaymeans (e.g., a display 65 shown in FIG. 4), the plurality of displaymeans being arranged in an array. The display control system includesdetermining means (e.g., a control unit 66 shown in FIG. 4 that performsthe processing of step S31 shown in FIG. 9) for determining whether ornot information of interest is being displayed; first recording controlmeans (e.g., a control unit 66 shown in FIG. 4 that performs theprocessing of step S34 shown in FIG. 9) for controlling recording of afirst image displayed on the display means when the determining meansdetermines that the information of interest is being displayed; secondrecording control means (e.g., a control unit 66 shown in FIG. 4 thatperforms the processing of step S42 shown in FIG. 10) for controllingrecording of a second image displayed on the display means when a userperforms a recording operation; and display control means (e.g., anoutput selection unit 63 shown in FIG. 4) for controlling display of thefirst image or the second image on the display means.

The display control system can further include a plurality of displaycontrol apparatuses (e.g., display devices 31-1 to 31-9 shown in FIG. 2)each associated with each of the plurality of display means andcontrolling display of an image on the display means. Each of theplurality of display control apparatuses can include the determiningmeans, the first recording control means, the second recording controlmeans and the display control means.

The display control system can further include a line-of-sight detectionapparatus (e.g., a line-of-sight detection apparatus 23 shown in FIG. 2)that detects a line of sight of the user, and the determining means ofeach of the plurality of display control apparatuses can determinewhether or not the information of interest is being displayed on thedisplay means on the basis of a detection result from the line-of-sightdetection apparatus.

The display control system can further include a remote controller(e.g., a remote controller 112 shown in FIG. 20) that controls theplurality of display control apparatuses, and the remote controller caninclude speech detecting means (e.g., an event determination unit 152shown in FIG. 29) for detecting specific speech of the user. Thedetermining means of each of the plurality of display controlapparatuses can determine whether or not the information of interest isbeing displayed on one of the plurality of display means on the basis ofa detection result from the speech detecting means.

The display control system can further include a remote controller(e.g., a remote controller 112 shown in FIG. 20) that controls theplurality of display control apparatuses, and the remote controller caninclude acceleration detecting means (e.g., an event determination unit123 shown in FIG. 22) for detecting an acceleration of the remotecontroller that is operated by the user. The determining means of eachof the plurality of display control apparatuses can determine whether ornot the information of interest is being displayed on one of theplurality of display means on the basis of a detection result from theacceleration detecting means.

According to the first embodiment of the present invention, there isprovided a display control method for a display control system forcontrolling display of images on a plurality of display means, theplurality of display means being arranged in an array. The displaycontrol method includes the steps of determining (e.g., step S31 shownin FIG. 9) whether or not information of interest is being displayed;controlling (e.g., step S34 shown in FIG. 9) recording of a first imagedisplayed on the display means when it is determined that theinformation of interest is being displayed; controlling (e.g., step S42shown in FIG. 10) recording of a second image displayed on the displaymeans when a user performs a recording operation; and controlling (e.g.,step S44 shown in FIG. 10) display of the first image or the secondimage on the display means.

According to a second embodiment of the present invention, there isprovided a display control apparatus (e.g., a display device 31-1 shownin FIG. 2) in a display control system (e.g., a multi-display systemshown in FIG. 2) including a plurality of display control apparatuses(e.g., display devices 31-1 to 31-9 shown in FIG. 2), for controllingdisplay of an image on display means (e.g., a display 65 shown in FIG.4) associated with the display control apparatus from among a pluralityof display means arranged in an array. The display control apparatusincludes the display means for displaying an image corresponding to animage signal input to the display control apparatus; determining means(e.g., a control unit 66 shown in FIG. 4 that performs the processing ofstep S31 shown in FIG. 9) for determining whether or not information ofinterest is being displayed; first recording control means (e.g., acontrol unit 66 shown in FIG. 4 that performs the processing of step S34shown in FIG. 9) for controlling recording of a first image displayed onthe display means when the determining means determines that theinformation of interest being displayed; second recording control means(e.g., a control unit 66 shown in FIG. 4 that performs the processing ofstep S42 shown in FIG. 10) for controlling recording of a second imagedisplayed on the display means when a user performs a recordingoperation; and display control means (e.g., an output selection unit 63shown in FIG. 4) for controlling display of the first image or thesecond image on the display means.

According to the second embodiment of the present invention, there isprovided a display control method or a program for a display controlapparatus in a display control system including a plurality of displaycontrol apparatuses, for controlling display of an image on displaymeans associated with the display control apparatus from among aplurality of display means arranged in an array. The display controlmethod or the program includes the steps of determining (e.g., step S31shown in FIG. 9) whether or not information of interest is beingdisplayed; controlling (e.g., step S34 shown in FIG. 9) recording of afirst image displayed on the display means associated with the displaycontrol apparatus when it is determined that the information of interestis being displayed; controlling (e.g., step S42 shown in FIG. 10)recording of a second image displayed on the display means associatedwith the display control apparatus when a user performs a recordingoperation; and controlling (e.g., step S44 shown in FIG. 10) display ofthe first image or the second image on the display means associated withthe display control apparatus.

Embodiments of the present invention will be described hereinafter withreference to the drawings.

FIG. 2 is a diagram showing an example of the structure of amulti-display system (the term system refers to a logical collection ofa plurality of apparatuses, and the apparatuses may or may not be placedin a housing) according to a first embodiment of the present invention.

In FIG. 2, the multi-display system includes a multi-display apparatus21 having a plurality of display devices 31-1 to 31-9, a remotecontroller 22, and a line-of-sight detection apparatus 23. In themulti-display system, the plurality of display devices 31-1 to 31-9operate in association with each other, as necessary, to display one ora plurality of images in different manners.

The multi-display apparatus 21 is configured to display images ofdifferent programs on the display devices 31-1 to 31-9 of which displayscreens are arranged in an array (or a matrix) of, for example, threerows and three columns or display an image of a program in an enlargedscale on the nine display devices 31-1 to 31-9 or on at least two of thedisplay devices 31-1 to 31-9 according to a command from the remotecontroller 22. In the following description, the display devices 31-1 to31-9 are referred to collectively as “display devices 31” orindividually as a “display device 31” unless separately specified. Thenumber of display devices in the multi-display apparatus 21 is notlimited to nine, and any number of rows and columns of display devicesmay be arranged horizontally and vertically, respectively.

The multi-display apparatus 21 further includes a light-receiving unit32 that receives light of a light signal into which a command sent fromthe remote controller 22 is converted. A command corresponding to thelight received by the light-receiving unit 32 is supplied to the displaydevices 31.

Each of the display devices 31 is provided with a tuner and a display.While a combination of the plurality of display devices 31 serves as themulti-display apparatus 21, each of the display devices 31 can alsoserve as a generally available stand-alone television receiver. Thedisplay devices 31-1 to 31-9 are (electrically) connected to each othervia a wired or wireless system (not shown), and each of the displaydevices 31-1 to 31-9 can operate solely or in conjunction with the otherdisplay devices 31.

The display devices 31 are configured to individually display images ofprograms desired by a user or collectively display an image of a singleprogram in an enlarged scale according to a command from the remotecontroller 22 received by the light-receiving unit 32. The displaydevices 31 have a zoom function and a memo function, and are configuredto zoom an image or images displayed thereon according to a command fromthe remote controller 22 for activating the zoom function (hereinafterreferred to as a “zoom command”) or to store an image or imagesdisplayed thereon according to a command from the remote controller 22for activating the memo function (hereinafter referred to as a “memocommand”) to display the stored image or images later.

Further, each of the display devices 31 determines whether or not a userwho is viewing an image displayed on the multi-display apparatus 21 haschanged his/her line of sight on the basis of line-of-sight detectionresult data from the line-of-sight detection apparatus 23. If the userhas changed his/her line of sight to an image displayed on one of thedisplay devices 31, the one display device 31 stores the image currentlydisplayed thereon and displays the stored image later.

In the display devices 31, therefore, images displayed thereon aretemporarily stored according to a command from the remote controller 22or a change in the user's line of sight.

The remote controller 22 includes an operation unit that is operated bya user. In response to an operation of the operation unit by a user, theremote controller 22 generates a command corresponding to the operation,and emits light to send the generated command in the form of apredetermined light signal.

The line-of-sight detection apparatus 23 includes a camera for capturingan eye or face image of a user who is viewing an image displayed on themulti-display apparatus 21, a computer for performing image processing,etc. An image signal of the eye image of the user captured by the camerais subjected to image processing to estimate the display device 31 towhich the user directs his/her line of sight, i.e., the image displayedon the display device 31 viewed by the user. The line-of-sight detectionapparatus 23 determines an identification (ID) number of the estimateddisplay device 31 as a viewing-screen number, and supplies thedetermined viewing-screen number to the display devices 31 asline-of-sight detection result data.

Examples of the line-of-sight detection method performed by theline-of-sight detection apparatus 23 may include, but not limited to,using a reflected infrared image, and using a visible light image. Theline-of-sight detection result data is not limited to a viewing-screennumber, and, for example, coordinate values of the detected line ofsight may be used as line-of-sight detection result data and supplied tothe display devices 31. In this case, each of the display devices 31performs processing, such as determining whether or not the line ofsight has changed, on the basis of the coordinate values.

In the example shown in FIG. 2, each of the display devices 31 includesa tuner and a display. The tuners and the displays may not be containedin the display devices 31. Alternatively, nine tuners may becollectively mounted in the multi-display apparatus 21, or a display maybe provided separately from the display devices 31 (in this case, thedisplay devices 31 serve as display control devices).

FIG. 3 is an external view showing an example of the structure of theremote controller 22 shown in FIG. 2.

In the example shown in FIG. 3, the remote controller 22 has on a frontsurface thereof a memo button 51, a zoom button 52, an Fn1 button 53, anFn2 button 54, screen selection buttons 55-1 to 55-9, and an operatingstick 56.

The memo button 51 is a button for a memo function for allowing thedisplay device 31 selected using the screen selection buttons 55-1 to55-9 to store an image displayed on the display device 31.

The zoom button 52 is a button for a zoom function for zooming anddisplaying an image displayed on the display device 31 selected usingthe screen selection buttons 55-1 to 55-9.

The Fn1 button 53 is a button for allowing an image stored according tothe line-of-sight detection result data of the line-of-sight detectionapparatus 23 to be displayed on the display device 31 selected using thescreen selection buttons 55-1 to 55-9. The Fn2 button 54 is a button forallowing an image stored according to a memo command to be selected anddisplayed on the display device 31 selected using the selection screenbuttons 55-1 to 55-9.

The screen selection buttons 55-1 to 55-9 are buttons for selectingimages (screens) displayed on the display devices 31-1 to 31-9,respectively. In the following description, the screen selection buttons55-1 to 55-9 are also referred to collectively as “screen selectionbuttons 55” or individually as a “screen selection button 55” unlessseparately specified. For example, when a single enlarged image isdisplayed on the plurality of display devices 31, the screen selectionbutton 55 associated with any of the display devices 31 on which theenlarged image is displayed is pressed to select the correspondingportion of the enlarged image.

The operating stick 56 is mounted on the front surface of the remotecontroller 22 so as to project therefrom and to be inclinable in360-degree directions with respect to the front surface of the remotecontroller 22. The operating stick 56 can be used to issue aninstruction in accordance with the inclined angle and direction of theoperating stick 56 to the display device 31 selected using the screenselection buttons 55-1 to 55-9 or to the multi-display apparatus 21.

For example, when the user presses any of the screen selection buttons55 on the remote controller 22, a screen selection command is generatedand transmitted to the display devices 31. One of the display devices 31associated with the pressed screen selection button 55 sets its screento a selected mode, and the other display devices 31 release theselected mode if the screens of the other display devices 31 are in theselected mode.

After the screen selection button 55 is selected, when the user pressesthe memo button 51, a memo command is generated and transmitted. Afterthe screen selection button 55 is selected, when the user presses thezoom button 52, a zoom command is generated and transmitted. After thescreen selection button 55 is selected, when the user presses the Fn1button 53 or the Fn2 button 54, a memory-image selection command isgenerated and transmitted. Those commands are received and executed bythe display device 31 that is associated with the screen selectionbutton 55 pressed by the user and that has recognized that its screen isin the selected mode.

In the example shown in FIG. 3, only relevant tools in the remotecontroller 22 shown in FIG. 2 are illustrated, and other tools such as apower button and tools for channel selection and volume control areomitted.

FIG. 4 is a diagram showing an example of the electrical structure ofthe display devices 31-1 to 31-9 shown in FIG. 2. In the example shownin FIG. 4, a section for processing audio signals is omitted.

The display device 31 shown in FIG. 4 includes a tuner 61, a memory 62,an output selection unit 63, an image processing unit 64, a display 65,a control unit 66, and a determination unit 67.

The tuner 61 demodulates a broadcast signal received from a broadcaststation via an antenna (not shown) using a predetermined method, andoutputs an image signal in the demodulated signal to the memory 62 andthe output selection unit 63. Thus, the memory 62 stores an image of theimage signal to be output to the display 65 via the output selectionunit 63 and the image processing unit 64. The memory 62 and the outputselection unit 63 receive not only the image signal from the tuner 61but also an image signal from an external recording apparatus orplayback apparatus.

The memory 62 is formed of, for example, a non-volatile memory or avolatile memory, and temporarily stores an image of an image signalinput from the tuner 61 and output to the display 65 under the controlof the control unit 66. For example, an image of an image signal inputat the time of a change in the user's line of sight and output to thedisplay 65 is stored in the memory 62 as a memory image #1, and an imageof an image signal input at the time of the reception of a memo commandand output to the display 65 is stored in the memory 62 as a memoryimage #2.

The output selection unit 63 selects the image signal input from thetuner 61 or the image signal corresponding to the memory image stored inthe memory 62 under the control of the control unit 66, and outputs theselected image signal to the image processing unit 64. The outputselection unit 63 further selects either the memory image #1 or memoryimage #2 stored in the memory 62 under the control of the control unit66 to read an image signal corresponding to the selected memory image,and outputs the read image signal to the image processing unit 64.

The image processing unit 64 performs predetermined image processing onthe image signal output from the output selection unit 63. The imageprocessing unit 64 also performs image zoom processing according to thecontrol from the control unit 66. The image signal subjected to theimage processing is output to the display 65. The display 65 displaysthe image corresponding to the image signal output from the imageprocessing unit 64. Therefore, the display 65 can also enlarge anddisplay the image corresponding to the image signal output from thetuner 61 or the image signal read from the memory 62.

Upon receiving a command from the remote controller 22 via thelight-receiving unit 32, the control unit 66 changes the selected modeof the screen of the display device 31, updates the memory image #2stored in the memory 62, selects an output image, or controls zoomparameters, etc., according to the received command.

If the received command is a screen selection command for selecting thescreen of the display device 31, the control unit 66 changes the screento the selected mode. If the received command is a screen selectioncommand for selecting any other screen and if the screen of the displaydevice 31 is in the selected mode, the control unit 66 releases theselected mode of the screen.

If the received command is a memo command, the control unit 66 updatesthe memory image #2 stored in the memory 62 with an image of an imagesignal input to the memory 62 at the time of the reception of the memocommand and output to the display 65, and further controls the outputselection unit 63 so as to read the image signal of the memory imagestored in the memory 62 (the memory image #1 or the memory image #2). Ifthe received command is a memory-image selection command, the controlunit 66 controls the output selection unit 63 so as to read the imagesignal of the memory image corresponding to the command. If the receivedcommand is a zoom command, the control unit 66 supplies the zoomparameters to the image processing unit 64, and controls the imageprocessing unit 64 so as to perform image zoom processing on the imagesignal.

The control unit 66 also controls to update the memory image #1 storedin the memory 62 in response to a notification of the time of a changein the user's line of sight from the determination unit 67.Specifically, the control unit 66 updates the memory image #1 stored inthe memory 62 with an image of an image signal input to the memory 62 atthe time of a change in the user's line of sight and output to thedisplay 65.

The determination unit 67 detects a change in the user's line of sighton the basis of the line-of-sight detection result data from theline-of-sight detection apparatus 23, and notifies the control unit 66of the time of the change in the user's line of sight.

FIG. 5 is a diagram showing an example of the electrical structure ofthe remote controller 22 shown in FIG. 2.

In the example shown in FIG. 5, the remote controller 22 includes anoperation unit 71, a control unit 72, a frame generation unit 73, atransmission unit 74, and a light-emitting unit 75.

The operation unit 71 includes the components shown in FIG. 3, i.e., thememo button 51, the zoom button 52, the Fn1 button 53, the Fn2 button54, the screen selection button 55, and the operating stick 56, andinputs an operation signal corresponding to a user's operation to thecontrol unit 72. The control unit 72 generates a command recognizable bythe display devices 31 on the basis of the operation signal input viathe operation unit 71, and outputs the generated command to the framegeneration unit 73.

The frame generation unit 73 generates a transmission frame based on thecommand output from the control unit 72, and transmits the transmissionframe to the transmission unit 74. The transmission unit 74 converts thetransmission frame generated by the frame generation unit 73 into alight-emission signal to cause the light-emitting unit 75 to emit light.

A process for recording an image using line-of-sight information will bedescribed with reference to FIGS. 6 and 7.

In an upper portion of FIG. 6, images G3 displayed on the display device31-3 (hereinafter also referred to as “screens G3 of the display device31-3) at times T1 and T2 (time T1<time T2) are illustrated.

As shown in a lower portion of FIG. 6, at a time T0 (time T0<time T1), auser A views a screen G2 of the display device 31-2 among screens G1 toG9 of the display devices 31-1 to 31-9. At the time T1, the user Achanges his/her line of sight to the screen G3 of the display device31-3. At the moment of the time T1 when the user A changes his/her lineof sight from the screen G2 to the screen G3, the user A notices thepresence of information of interest (hereinafter also referred to as an“object”) on the screen G3.

To record the object being displayed on the screen G3 (e.g., thepitcher's grip before throwing the ball when the pitcher cocks his arm),for example, the user A presses the screen selection button 55-3 on theremote controller 22 to select the screen G3 of the display device 31-3,and then presses the memo button 51 at the time T2. However, theinformation on the screen G3 has already changed at the moment of thetime T2 when the user A presses the memo button 51, and the user A mayfail to record the object displayed at the moment of the time T1 on thescreen G3. In the example shown in FIG. 6, the screen G3 displayed atthe time T2 shows an image of a moment when the pitcher throws the ball.

Therefore, as shown in FIG. 6, the display device 31-3 estimates themoment when the user A changes his/her line of sight to the screen G3 tobe the moment when the user A desires to make a memo of (or record) thescreen G3, and records an image displayed on the screen G3 at the momentwhen the user's line of sight changes (i.e., the screen G3 displayed atthe time T1) in the memory 62 in advance. Then, the image recorded inthe memory 62 is displayed on the screen G3 at the time T2 when the userA actually presses the memo button 51.

This feature will be specifically described with reference to FIG. 7. Inthe example shown in FIG. 7, at the time T0, the user A directs his/herline of sight to the screen G2 of the display device 31-2 among thescreens G1 to G9 of the display devices 31-1 to 31-9. At the time T1,the user A changes his/her line of sight from the screen G2 to thescreen G3 of the display device 31-3. In FIG. 7, a change in the screenviewed by the user A over time, and a change in the output imagedisplayed on the screen G3 over time are also illustrated.

At the time T1 when the screen viewed by the user A is changed from thescreen G2 to the screen G3 based on the viewing-screen number outputfrom the line-of-sight detection apparatus 23, i.e., when the user Achanges his/her line of sight from the screen G2 to the screen G3, thedisplay device 31-3 stores the information displayed at the time T1 onthe screen G3 in the memory 62 as a memory image #1. At the time T2 whenthe user A actually presses the memo button 51, the display device 31-3stores the information displayed at the time T2 on the screen G3 in thememory 62 as a memory image #2.

Further, as indicated by the output image on the screen G3, at the timeT2, the display device 31-3 switches an output image on the screen G3from the image output from the tuner 61 to the memory image #1. Apresetting may be performed so as to switch an output image on thescreen G3 to the memory image #2 instead of the memory image #1.

Accordingly, the moment when the user A moves his/her line of sight isestimated to be the moment when the user A desires to make a memo, andan image displayed when a change in the line of sight of the user A isdetected is recorded. Therefore, for example, even if the user Ahesitates to operate the remote controller 22, failure to catch anobject of interest can be avoided.

A process of the display device 31 for recording an image usingline-of-sight information will be described in detail with reference toa flowchart shown in FIG. 8.

For example, the power of the multi-display apparatus 21 is turned on,and the tuner 61 of the display device 31 demodulates a broadcast signalreceived from a broadcast station via the antenna (not shown) using apredetermined method, and outputs an image signal in the demodulatedsignal to the memory 62 and the output selection unit 63. The outputselection unit 63 selects the image signal input from the tuner 61 underthe control of the control unit 66, and outputs the selected imagesignal to the image processing unit 64. The image processing unit 64performs predetermined image processing on the image signal output fromthe output selection unit 63, and outputs the resulting image signal tothe display 65. The display 65 displays an image corresponding to theimage signal output from the image processing unit 64.

In step S11, the control unit 66 checks a preset dual in-line package(DIP) switch or the like to obtain the ID number of the display device31. The ID number may be obtained by, instead of using the DIP switch,communicating with the other display devices 31.

A user is viewing an image displayed on the display 65 of any of thedisplay devices 31-1 to 31-9 (e.g., an image displayed on the displaydevice 31-3, i.e., the screen G3 of the display device 31-3).

The line-of-sight detection apparatus 23 captures an eye or face imageof a user who is viewing an image displayed on the multi-displayapparatus 21, and performs image processing on the captured eye image ofthe user to estimate the display device 31 to which the user directshis/her line of sight, i.e., the screen of the display device 31 viewedby the user. The line-of-sight detection apparatus 23 determines the IDnumber of the estimated display device 31 as a viewing-screen number,and supplies the determined viewing-screen number to the display devices31 as line-of-sight detection result data.

Since the line of sight is unstable, a viewing-screen number of a screenis determined if it is determined that the screen is viewed over severalconsecutive samples. If it is estimated that none of the screens of thedisplay devices 31 is viewed by the user, viewing-screen number “−1” issupplied to the display devices 31 as line-of-sight detection resultdata. For example, if the viewing-screen number is changed from the IDnumber of any of the display devices 31 to the viewing-screen number“−1”, no image is stored in the memory 62 of any of the display devices31.

In step S12, the determination unit 67 obtains the line-of-sightdetection result data from the line-of-sight detection apparatus 23 todetect a change in the user's line of sight on the basis of theline-of-sight detection result data obtained from the line-of-sightdetection apparatus 23, and notifies the control unit 66 of the time ofthe change in the user's line of sight. The control unit 66 is alsonotified of the ID number of the line-of-sight destination.

In step S13, the control unit 66 performs a memory control process. Thememory control process will be described with reference to a flowchartshown in FIG. 9.

In step S31, the control unit 66 determines whether or not a change inthe line of sight has occurred. If the control unit 66 is notified ofthe time of a change in the user's line of sight by the determinationunit 67, it is determined in step S31 that a change in the line of sighthas occurred, and the process proceeds to step S32.

In step S32, the control unit 66 determines whether or not the line ofsight has been changed to the screen of the display device 31 on thebasis of the ID number of the line-of-sight destination notified by thedetermination unit 67. If there is a match between the ID number of theline-of-sight destination notified by the determination unit 67 and theID number of the display device 31 obtained in step S11 shown in FIG. 8,it is determined in step S32 that the line of sight has been changed tothe screen of the display device 31, and the process proceeds to stepS33.

In step S33, the control unit 66 determines whether or not the memory 62is in a protected mode. If it is determined that the memory 62 is not inthe protected mode, i.e., the protected mode of the memory 62 has beenreleased, the process proceeds to step S34.

For example, an image recorded in the memory 62 at the moment when theuser A changes his/her line of sight, which is estimated to be themoment when the user A desires to make a memo, may be overwritten by anaccidental action of the user A. For example, the user A mayaccidentally move his/her line of sight during a zoom operation of theimage, and an image displayed at the time of the action may be recordedin the memory 62 to overwrite the previous image.

A user's operation of pressing the memo button 51 can be considered as aclearly intended operation to make a memo of a current image. Therefore,when the user operates the memo button 51 (i.e., upon receiving a memocommand from the remote controller 22), the display device 31 sets theimage stored in the memory 62 to the protected (or overwrite prohibited)mode (step S43 shown in FIG. 10, as described below). If the user'soperation of the remote controller 22 is not performed for apredetermined period of time or longer, the display device 31 releasesthe protected mode of the memory 62 (step S22 shown in FIG. 8).

Therefore, if the memory 62 is not in the protected mode, in step S34,the control unit 66 stores the image of the image signal input to thememory 62 as a memory image #1. Specifically, the control unit 66updates the memory image #1 stored in the memory 62 with an image of animage signal input to the memory 62 at the time of the change in theuser's line of sight and output to the display 65. Then, the processreturns to step S13 shown in FIG. 8 and proceeds to step S14.

If it is determined in step S31 that no change in the line of sight hasoccurred, or if it is determined in step S32 that the line of sight hasnot been changed to the screen of the display device 31, or if it isdetermined in step S33 that the memory 62 is in the protected mode, theprocess also returns to step S13 shown in FIG. 8 and proceeds to stepS14.

Referring back to FIG. 8, in step S14, the control unit 66 determineswhether or not a command has been received from the remote controller22. If it is determined that a command has been received from the remotecontroller 22, in step S15, it is determined whether or not the receivedcommand is a screen selection command.

If it is determined in step S15 that the received command is a screenselection command, in step S16, the control unit 66 changes the selectedmode of the screen of the display device 31 according to the screenselection command. For example, if the received command is a screenselection command for selecting the screen of the display device 31, thecontrol unit 66 changes the screen to the selected mode. If the receivedcommand is a screen selection command for selecting the screen of anyother display device 31 and if the screen of the display device 31 is inthe selected mode, the control unit 66 releases the selected mode. Then,the process returns to step S14, and the subsequent processing isrepeated.

If it is determined in step S15 that the received command is not ascreen selection command, the process skips step S16. In step S17, thecontrol unit 66 determines whether or not the screen of the displaydevice 31 is in the selected mode. If it is determined in step S17 thatthe screen is in the selected mode, in step S18, the control unit 66resets a counter value.

The control unit 66 includes an internal counter (not shown) formeasuring a count of the occurrence of the event upon which a commanddirected to the display device 31 has not been received from the user.Since it is determined in step S17 that the screen is in the selectedmode, i.e., the command received from the remote controller 22 is acommand directed to the display device 31, in step S18, the value of thecounter is reset.

In step S19, the control unit 66 performs a process in accordance withthe command. The process in accordance with the command will bedescribed with reference to a flowchart shown in FIG. 10.

In step S41, the control unit 66 determines whether or not the receivedcommand is a memo command. If it is determined that the received commandis a memo command, in step S42, the control unit 66 stores the image ofthe image signal input to the memory 62 as a memory image #2.Specifically, the control unit 66 updates the memory image #2 stored inthe memory 62 with an image of an image signal input to the memory 62 atthe time of the reception of the memo command and output to the display65. Then, the process proceeds to step S43.

In step S43, the control unit 66 sets the memory 62 to the protectedmode. Thus, the memory image #1 and memory image #2 stored in the memory62 are prohibited from being overwritten until it is determined that theuser's operation of the remote controller 22 is not performed for apredetermined period of time or longer.

In step S44, the control unit 66 controls the selection of the outputselection unit 63 so as to read the image signal of the memory image #1stored in the memory 62. That is, the output selection unit 63 selectsthe memory image #1 stored in the memory 62 to read the image signal ofthe selected memory image, and outputs the read image signal to theimage processing unit 64. The image signal subjected to image processingby the image processing unit 64 is output to the display 65. The display65 displays the memory image #1 corresponding to the image signal outputfrom the image processing unit 64.

A presetting may be preformed so as to display the memory image #2 instep S44.

If it is determined in step S41 that the received command is not a memocommand, the process skips steps S42 to S44 and proceeds to step S45. Instep S45, the control unit 66 determines whether or not the receivedcommand is a memory-image selection command. If it is determined thatthe received command is a memory-image selection command, in step S46,the control unit 66 controls the output selection unit 63 so as to readthe image signal of the memory image in accordance with the command (thememory image #1 or the memory image #2).

Specifically, if the received command is a memory-image selectioncommand generated by pressing the Fn1 button 53, the output selectionunit 63 selects the memory image #1 stored in the memory 62 to read theimage signal of the selected memory image #1, and outputs the read imagesignal to the image processing unit 64. If the received command is amemory-image selection command generated by pressing the Fn2 button 54,the output selection unit 63 selects the memory image #2 stored in thememory 62 to read the image signal of the selected memory image #2, andoutputs the read image signal to the image processing unit 64.

The image signal subjected to image processing by the image processingunit 64 is output to the display 65. The display 65 displays the memoryimage #1 or memory image #2 corresponding to the image signal from theimage processing unit 64.

If it is determined in step S45 that the received command is not amemory-image selection command, the process skips step S46 and proceedsto step S47. In step S47, the control unit 66 determines whether or notthe received command is a zoom command. If it is determined that thereceived command is a zoom command, the process proceeds to step S48.

In step S48, the control unit 66 sends zoom parameters (such as zoommagnification and zoom range) in accordance with the command to theimage processing unit 64, and controls the image processing unit 64 soas to perform image zoom processing on the image signal. Specifically,the image processing unit 64 performs predetermined image processing onthe image signal output from the output selection unit 63, and performsimage zoom processing on the image signal on the basis of the zoomparameters sent from the control unit 66. The resulting image signal isoutput to the display 65, and the display 65 displays an imagecorresponding to the image signal subjected to the zoom processing onthe basis of the zoom parameters.

If it is determined in step S47 that the received command is not a zoomcommand, the process skips step S48, and returns to step S19 shown inFIG. 8. Then, the process further returns to step S12, and thesubsequent processing is repeated.

If it is determined in step S14 shown in FIG. 8 that no command has beenreceived from the remote controller 22, or if it is determined in stepS17 that the screen is not in the selected mode, the process proceeds tostep S20. Since no command directed to the display device 31 has beenreceived, in step S20, the control unit 66 increments the value of theinternal counter.

In step S21, the control unit 66 determines whether or not the value ofthe counter is equal to or more than a predetermined threshold value. Ifit is determined that the value of the counter is equal to or more thanthe predetermined threshold value, i.e., when a period of time not lessthan the threshold value has elapsed since the display device 31received the last command, in step S22, the control unit 66 releases theprotected mode of the memory 62. Therefore, a write to the memory 62 ispermitted in accordance with the line-of-sight detection result data orthe memo command.

If it is determined in step S21 that the value of the counter is lessthan the threshold value, the process skips step S22 and returns to stepS12, and the subsequent processing is repeated.

Accordingly, the moment when the user moves his/her line of sight isestimated to be the moment when the user desires to make a memo, and animage displayed on the line-of-sight destination is recorded in thememory 62 when the user changes his/her line of sight. Therefore, forexample, even if the user hesitates to operate the remote controller 22,failure to catch an object of interest can be avoided.

Further, an image displayed on the display device 31 as a target to beoperated (set to the selected mode) when a memo command is received isalso recorded in the memory 62 separately from the image displayed onthe line-of-sight destination. Therefore, an image displayed when theuser successfully operates the remote controller 22 can also beobtained.

Further, if there exists a memory image stored in the memory 62according to a memo command, the memory 62 is set to the protected modeso as to prevent the memory image from being overwritten. This canprevent the memory image stored in the memory 62 from being deleted by auser's accidental action such as moving his/her line of sight during theoperation of the remote controller 22.

A process of the remote controller 22 corresponding to the process shownin FIG. 8 will now be described with reference to a flowchart shown inFIG. 11.

For example, the user operates the component of the operation unit 71shown in FIG. 3, such as the memo button 51, the zoom button 52, the Fn1button 53, the Fn2 button 54, the screen selection button 55, or theoperating stick 56. The control unit 72 generates a command recognizableby the display devices 31 on the basis of the operation signal input viathe operation unit 71, and outputs the generated command to the framegeneration unit 73.

In step S51, the frame generation unit 73 waits for a command to beinput. If it is determined that a command has been input, in step S52,the frame generation unit 73 generates a transmission frame based on thecommand from the control unit 72, and transmits the transmission frameto the transmission unit 74. In step S53, the transmission unit 74converts the transmission frame generated by the frame generation unit73 into a light-emission signal to cause the light-emitting unit 75 toemit light. Thus, the light-receiving unit 32 of the display device 31receives the light from the remote controller 22, and transmits acommand corresponding to the received light to the display devices 31.

FIG. 12 is an external view showing another example of the structure ofthe remote controller 22 shown in FIG. 2. Unlike the remote controller22 shown in FIG. 3, the screen selection buttons 55-1 to 55-9 are notprovided on a front surface of the remote controller 22 shown in FIG.12. As in the remote controller 22 shown in FIG. 3, the remotecontroller 22 shown in FIG. 12 includes on the front surface thereof amemo button 51, a zoom button 52, an Fn1 button 53, an Fn2 button 54,and an operating stick 56.

In the remote controller 22 shown in FIG. 12, therefore, instead ofusing the screen selection buttons 55-1 to 55-9, the user operates theoperating stick 56 in the vertical, horizontal, or diagonal direction tomove a cursor K for selecting one of the screens G1 to G9 of the displaydevices 31-1 to 31-9 to a desired screen, thereby selecting one of thescreens G1 to G9.

In the example shown in FIG. 12, the operating stick 56 is inclined tothe right in FIG. 12 to move the cursor K placed on the screen G4 to thescreen G5 (to the right in FIG. 12).

Accordingly, the tool for selecting a screen is not limited to thescreen selection buttons 55-1 to 55-9 shown in FIG. 3, and may beimplemented by the operating stick 56 shown in FIG. 12.

A process for recording a moving image, instead of a still image, usingline-of-sight information will be described with reference to FIG. 13.

In the foregoing description, to address a situation where a user findsan object to make a memo of at the moment when the user moves his/herline of sight from a certain screen to another screen, two images, thatis, an image displayed when the user moves his/her line of sight and animage displayed when the user operates the memo button 51, are recorded.In some cases, however, an object of which a user desires to make a memomay appear for a period of time during which the user is moving his/herline of sight and viewing through screens, rather than at the momentwhen the user moves his/her line of sight from a certain screen toanother screen. In such cases, two images may be insufficient tosuccessfully record the object.

In such cases, if a sufficient memory capacity is available, a sequenceof images displayed for a period of time after the user has changedhis/her line of sight and before the user operates the memo button 51,rather than an image displayed when the user changes his/her line ofsight, is recorded as a moving image. Therefore, the user can obtainimages including a desired object. Specifically, in such cases, apredetermined period of time after the moment when the user changeshis/her line of sight, rather than the moment when the user changeshis/her line of sight, is estimated to be a period of time in whichinformation of interest may be shown, and image signals occurring duringthat period of time are recorded in the memory 62 as a moving image sothat a desired frame of the moving image can be selected later byoperating the remote controller 22.

In the example shown in FIG. 13, as in the example shown in FIG. 7, at atime T0, the user A directs his/her line of sight to the screen G2 ofthe display device 31-2 among the screens G1 to G9 of the displaydevices 31-1 to 31-9. At time T1, the user A changes his/her line ofsight from the screen G2 to the screen G3 of the display device 31-3. InFIG. 13, a change in the screen viewed by the user A over time is alsoillustrated.

If a change in the screen viewed by the user A to the screen G3 from thescreen G2 is detected on the basis of the viewing-screen number outputfrom the line-of-sight detection apparatus 23, i.e., if the user Achanges his/her line of sight from the screen G2 to the screen G3, thedisplay device 31-3 stores in the memory 62, as a moving image,information displayed on the screen G3 for a time period W starting fromthe time T1 when the user A changes his/her line of sight from thescreen G2 to the screen G3 to, for example, a time T2 when the user Aactually presses the memo button 51.

Therefore, for example, even if the user A finds the presence ofinformation of interest on the screen G3 at a time T12 (time T1<timeT12<time T2) at which a certain time has elapsed since the time T1 whenthe user A changed his/her line of sight from the screen G2 to thescreen G3, the user A can check the stored moving image later to viewthe desired information.

Accordingly, a predetermined period of time after the moment when theuser A changes his/her line of sight is estimated to be a period of timein which information of which the user A desires to make a memo isshown, and a moving image displayed during that period of time isrecorded. Therefore, for example, even if the user A hesitates tooperate the remote controller 22, failure to catch an object of interestcan be avoided. The predetermined period of time can be specified, butit depends on the remaining capacity of the memory 62.

A process for recording a moving image using line-of-sight informationby the display device 31 in the example shown in FIG. 13 will bedescribed in detail with reference to a flowchart shown in FIG. 14. Theprocess shown in FIG. 14 is different from that shown in FIG. 8 in theprocessing of steps S73 and S74. The remaining processing of steps S71,S72, and S75 to S83 is basically similar to the processing of steps S11,S12, and S14 to S22 shown in FIG. 8, respectively, and a detaileddescription thereof is thus omitted to avoid redundancy.

In step S71, the control unit 66 checks a preset DIP switch or the liketo obtain the ID number of the display device 31.

The line-of-sight detection apparatus 23 captures an eye or face imageof a user who is viewing an image displayed on the multi-displayapparatus 21, and performs image processing on the captured eye image ofthe user to estimate the display device 31 to which the user directshis/her line of sight, i.e., the screen of the display device 31 viewedby the user. The line-of-sight detection apparatus 23 determines the IDnumber of the estimated display device 31 as a viewing-screen number,and supplies the determined viewing-screen number to the display devices31 as line-of-sight detection result data.

In step S72, the determination unit 67 obtains the line-of-sightdetection result data from the line-of-sight detection apparatus 23 todetect a change in the user's line of sight on the basis of theline-of-sight detection result data obtained from the line-of-sightdetection apparatus 23, and notifies the control unit 66 of the time ofthe change in the user's line of sight. The control unit 66 is alsonotified of the ID number of the line-of-sight destination.

In step S73, the control unit 66 performs a memory control process. Thememory control process will be described with reference to a flowchartshown in FIG. 15. The process shown in FIG. 15 is different from thatshown in FIG. 9 in the processing of step S94. The remaining processingof steps S91 to S93 is basically similar to the processing of steps S31to S33 shown in FIG. 9, respectively, and a detailed description thereofis thus omitted to avoid redundancy.

In step S91, the control unit 66 determines whether or not a change inthe line of sight has occurred. If the control unit 66 is notified ofthe time of a change in the user's line of sight by the determinationunit 67, it is determined in step S91 that a change in the line of sighthas occurred, and the process proceeds to step S92.

In step S92, the control unit 66 determines whether or not the line ofsight has been changed to the screen of the display device 31 on thebasis of the ID number of the line-of-sight destination notified by thedetermination unit 67. If there is a match between the ID number of theline-of-sight destination notified by the determination unit 67 and theID number of the display device 31 obtained in step S71 shown in FIG.14, it is determined in step S92 that the line of sight has been changedto the screen of the display device 31, and the process proceeds to stepS93.

In step S93, the control unit 66 determines whether or not the memory 62is in a protected mode. If it is determined that the memory 62 is not inthe protected mode, i.e., the protected mode of the memory 62 has beenreleased, the process proceeds to step S94.

In step S94, the control unit 66 turns on a memory-write flag for thememory 62. Specifically, the control unit 66 sets a write destinationpointer to a start address of the memory 62 for image recording, andstarts the recording of the input image signal. Then, the processreturns to step S73 shown in FIG. 14 and proceeds to step S74.

If it is determined in step S91 that no change in the line of sight hasoccurred, or if it is determined in step S92 that the line of sight hasnot been changed to the screen of the display device 31, or if it isdetermined in step S93 that the memory 62 is in the protected mode, theprocess also returns to step S73 shown in FIG. 14 and proceeds to stepS74.

Referring back to FIG. 14, in step S74, the memory 62 performs a memorywrite process according to the memory control process in step S73. Thememory write process will be described with reference to a flowchartshown in FIG. 16.

In step S101, the memory 62 determines whether or not the memory-writeflag is turned on. If it is determined that the memory-write flag isturned on, the process proceeds to step S102. In step S102, the memory62 determines whether or not a sufficient write capacity remains.

If it is determined in step S102 that a sufficient write capacityremains, in step S103, the memory 62 records the input image signal.Therefore, image signals occurring during a period of time after a startinstruction of recording is given in step S94 shown in FIG. 15 until astop instruction of recording is given in step S112 shown in FIG. 17, asdescribed below, are recorded as a memory moving image.

In general, when the memory-write flag is turned on, the memory is notin the protected mode, and the previous moving image is overwritten.However, if the time until a stop instruction of recording is given issignificantly long, or if no memo command is received and no stopinstruction of recording is given, the recording of a moving image isstopped when the write capacity has exhausted.

If it is determined in step S101 that the memory-write flag is turnedoff, or if it is determined in step S102 that no sufficient writecapacity remains, the process skips step S103. Then, the process returnsto step S74 shown in FIG. 14 and proceeds to step S75. In this case, noimage signals are recorded.

Referring back to FIG. 14, in step S75, the control unit 66 determineswhether or not a command has been received from the remote controller22. If it is determined that a command has been received from the remotecontroller 22, in step S76, it is determined whether or not the receivedcommand is a screen selection command.

If it is determined in step S76 that the received command is a screenselection command, in step S77, the control unit 66 changes the selectedmode of the screen of the display device 31 according to the screenselection command. Specifically, if the received command is a screenselection command for selecting the screen of the display device 31, thecontrol unit 66 changes the screen to the selected mode. If the receivedcommand is a screen selection command for selecting the screen of anyother display device 31 and if the screen of the display device 31 is inthe selected mode, the control unit 66 releases the selected mode. Then,the process returns to step S75, and the subsequent processing isrepeated.

If it is determined in step S76 that the received command is not ascreen selection command, in step S78, the control unit 66 determineswhether or not the screen of the display device 31 is in the selectedmode. If it is determined in step S78 that the screen is in the selectedmode, in step S79, the control unit 66 resets the value of the counter.

In step S80, the control unit 66 performs a process in accordance withthe command. The process in accordance with the command will bedescribed with reference to a flowchart shown in FIG. 17. In FIG. 17,the processing of steps S111, S113, S114, S116, and S117 is basicallysimilar to the processing of steps S41, S43, S45, S47, and S48 shown inFIG. 10, respectively, and a detailed description thereof is thusomitted to avoid redundancy.

In step S111, the control unit 66 determines whether or not the receivedcommand is a memo command. If it is determined that the received commandis a memo command, in step S112, the control unit 66 turns off thememory-write flag. Specifically, the control unit 66 stops the recordingin the memory 62 instructed to be started in step S94 shown in FIG. 15.

In step S113, the control unit 66 sets the memory 62 to the protectedmode. Thus, the moving image stored in the memory 62 is prohibited frombeing overwritten until it is determined that the user's operation ofthe remote controller 22 is not performed for a predetermined period oftime or longer. A moving image recorded if no memo command is receivedis not prohibited from being overwritten, and is therefore overwrittennext time the user changes his/her line of sight.

If it is determined in step S111 that the received command is not a memocommand, the process skips steps S112 and S113, and proceeds to stepS114. In step S114, the control unit 66 determines whether or not thereceived command is a memory-image selection command. If it isdetermined that the received command is a memory-image selectioncommand, in step S115, the control unit 66 controls the output selectionunit 63 so as to read the image signal of the memory image in accordancewith the command from the memory moving image.

As shown in FIG. 18, the remote controller 22 may be configured torecord a moving image. In this case, the remote controller 22 includeson a front surface thereof, a memo button 51, a zoom button 52, screenselection buttons 55-1 to 55-9, and an operating stick 56, as in theremote controller 22 shown in FIG. 3. The remote controller 22 shown inFIG. 18 further includes on the front surface thereof, in place of theFn1 button 53 and Fn2 button 54 shown in FIG. 3, an image selectionslider 81 for selecting a desired memory image (frame) from the memorymoving image.

In the remote controller 22 shown in FIG. 18, after the screen selectionbutton 55 is selected, when the user moves a knob on the image selectionslider 81 to the right or left, an image selection command for selectinga memory image at a position in the memory moving image corresponding tothe position of the knob on the image selection slider 81 is generatedand transmitted.

Therefore, in the case of a memory-image selection command generated bymoving the image selection slider 81, the output selection unit 63selects an image signal of a memory image located at a position,specified by the command, in the memory moving image stored in thememory 62 to read the image signal of the selected memory image, andoutputs the read image signal to the image processing unit 64.

The image signal subjected to image processing by the image processingunit 64 is output to the display 65. The display 65 displays a memoryimage corresponding to the image signal output from the image processingunit 64.

If it is determined in step S114 that the received command is not amemory-image selection command, the process skips step S115 and proceedsto step S116. In step S116, the control unit 66 determines whether ornot the received command is a zoom command. If it is determined that thereceived command is a zoom command, the process proceeds to step S117.

In step S117, the control unit 66 sends zoom parameters (such as zoommagnification and zoom range) in accordance with the command to theimage processing unit 64, and controls the image processing unit 64 soas to perform image zoom processing on the image signal. Specifically,the image processing unit 64 performs predetermined image processing onthe image signal output from the output selection unit 63, and performsimage zoom processing on the image signal on the basis of the zoomparameters sent from the control unit 66. The resulting image signal isoutput to the display 65, and the display 65 displays an imagecorresponding to the image signal subjected to the zoom processing onthe basis of the zoom parameters.

If it is determined in step S116 that the received command is not a zoomcommand, the process skips step S117, and returns to step S80 shown inFIG. 14. Then, the process further returns to step S72, and thesubsequent processing is repeated.

If it is determined in step S75 shown in FIG. 14 that no command hasbeen received from the remote controller 22, or if it is determined instep S78 that the screen of the display device 31 is not in the selectedmode, the process proceeds to step S81. Since no command directed to thedisplay device 31 of the display device 31 has been received, in stepS81, the control unit 66 increments the value of the internal counter.

In step S82, the control unit 66 determines whether or not the value ofthe counter is equal to or more than a predetermined threshold value. Ifit is determined that the value of the counter is equal to or more thanthe predetermined threshold value, i.e., when a period of time not lessthan the threshold value has elapsed since the display device 31received the last command, in step S83, the control unit 66 releases theprotected mode of the memory 62. Therefore, a write to the memory 62 ispermitted in accordance with the line-of-sight detection result data orthe memo command.

If it is determined in step S82 that the value of the counter is lessthan the threshold value, the process skips step S83 and returns to stepS72, and the subsequent processing is repeated.

Accordingly, a predetermined period of time after the moment when theuser moves his/her line of sight is estimated to be a period of time inwhich information of which the user desires to make a memo is shown. Ifa change in the user's line of sight has occurred, a moving imagedisplayed on the line-of-sight destination is recorded in the memory 62.Therefore, for example, even if the user hesitates to operate the remotecontroller 22, failure to catch an object of interest can further beavoided.

FIG. 19 is an external view showing another example of the structure ofthe remote controller 22 shown in FIG. 18. The remote controller 22shown in FIG. 19 includes on a front surface thereof, in place of theimage selection slider 81, a jump-to-first button 91 for returning tothe first frame, a return-to-previous button 92 for returning to theprevious frame, a go-to-next button 93 for proceeding to the next frame,and a jump-to-last button 94 for jumping to the last frame, which isdifferent from the remote controller 22 shown in FIG. 18. The remotecontroller 22 shown in FIG. 19 further includes on the front surfacethereof a memo button 51, a zoom button 52, screen selection buttons55-1 to 55-9, and an operating stick 56, as in the remote controller 22shown in FIG. 18.

In the remote controller 22 shown in FIG. 19, after the screen selectionbutton 55 is selected, when the user presses the jump-to-first button91, the return-to-previous button 92, the go-to-next button 93, or thejump-to-last button 94, an image selection command for selecting amemory image (frame) associated with the pressed button from apredetermined memory image in the memory moving image is generated andtransmitted.

For example, when the user presses the go-to-next button 93, the outputselection unit 63 selects an image signal of a memory image (frame) nextto the predetermined memory image of the memory moving image stored inthe memory 62 to read the image signal of the selected memory image, andoutputs the read image signal to the image processing unit 64.Therefore, the display 65 displays a memory image corresponding to theimage signal output from the image processing unit 64.

FIG. 20 is a diagram showing a structure of a multi-display systemaccording to a second embodiment of the present invention. Themulti-display system shown in FIG. 20 is different from themulti-display system shown in FIG. 2 in that the multi-display systemshown in FIG. 20 includes a remote controller 112 in place of the remotecontroller 22 and does not include the line-of-sight detection apparatus23. As in the multi-display system shown in FIG. 2, the multi-displaysystem shown in FIG. 20 further includes a multi-display apparatus 21′including a plurality of display devices 31′-1 to 31′-9 (the displaydevices 31′-1 to 31′-9 are hereinafter referred to collectively as“display devices 31′” or individually as a “display device 31′” unlessseparately specified).

In the multi-display system shown in FIG. 20, when a movement (a changein acceleration) of the remote controller 112 caused by a user'soperation is detected instead of a change in the user's line of sight onthe basis of the line-of-sight detection result data from theline-of-sight detection apparatus 23, each of the display devices 31′-1to 31′-9 stores an image currently displayed thereon. The movement ofthe remote controller 112 is detected when the light-receiving unit 32receives an event-determination result flag from the remote controller112 and supplies the flag to the display devices 31′.

The remote controller 112 includes an operation unit 71 having, as inthe remote controller 22 shown in FIG. 3, the memo button 51, zoombutton 52, Fn1 button 53, Fn2 button 54, screen selection button 55, andoperating stick 56 shown in FIG. 3, and an acceleration sensor 121 (seeFIG. 22, as described below). Upon detecting, in addition to thecommands described above with reference to FIG. 3, a rapid change inacceleration from a result of an output value of the acceleration sensor121 converted from analog to digital (AD) form, the remote controller112 generates an event-determination result flag, and emits light tosend the generated event-determination result flag in the form of apredetermined light signal.

The remote controller 112 may transmit an event-determination resultflag via infrared communication for use in normal command transmission,or using a wireless local area network (LAN), Bluetooth™, or the like.The transmission via directional infrared communication is more robustbecause an event-determination result causing noise is less likely to bereceived in the operation of the remote controller 112 that is notdirected to the front of the multi-display apparatus 21′.

FIG. 21 is a diagram showing an example of the electrical structure ofthe display device 31′ shown in FIG. 20. The display device 31′ shown inFIG. 21 is different from the display device 31 shown in FIG. 4 in thatthe determination unit 67 is not provided. As in the display device 31shown in FIG. 4, the display device 31′ includes a tuner 61, a memory62, an output selection unit 63, an image processing unit 64, a display65, and a control unit 66.

Upon receiving data from the remote controller 112 via thelight-receiving unit 32, the control unit 66 shown in FIG. 21 determineswhether or not the received data is an event-determination result flag.If the received data is an event-determination result flag, i.e., if theremote controller 112 detects a rapid change in acceleration, thecontrol unit 66 controls to update the memory image #1 stored in thememory 62. If the data received from the remote controller 112 is acommand, the control unit 66 updates the memory image #2 stored in thememory 62, selects an output image, or control zoom parameters, etc.,according to the command in a manner described above with reference toFIG. 4.

FIG. 22 is a diagram showing an example of the electrical structure ofthe remote controller 112 shown in FIG. 21. The remote controller 112shown in FIG. 22 is different from the remote controller 22 shown inFIG. 5 in that an acceleration sensor 121, an AD conversion unit 122,and an event determination unit 123 are additionally provided. As in theremote controller 22 shown in FIG. 5, the remote controller 112 shown inFIG. 22 further includes an operation unit 71, a control unit 72, aframe generation unit 73, a transmission unit 74, and a light-emittingunit 75. In the multi-display system shown in FIG. 20, therefore, thedetermination unit 67 included in the display device 31 in themulti-display system shown in FIG. 2 is included in the remotecontroller 112 as the event determination unit 123.

As well as outputting a generated command to the frame generation unit73, the control unit 72 shown in FIG. 22 generates anevent-determination result flag when an event-detection flag from theevent determination unit 123 is turned on, and outputs theevent-determination result flag to the frame generation unit 73.

The acceleration sensor 121 detects an acceleration of the remotecontroller 112 caused by, for example, the remote controller 112 held bythe user, and outputs acceleration-sensor data to the AD conversion unit122. The AD conversion unit 122 converts the output value from theacceleration sensor 121 (acceleration-sensor data) from analog todigital (AD) form, and outputs the AD-converted acceleration-sensor datato the event determination unit 123.

Upon detecting a rapid change in acceleration from the AD-convertedacceleration-sensor data, the event determination unit 123 turns on anevent-detection flag, and outputs the turned on event-detection flag tothe control unit 72. If no rapid change in acceleration is detected, theevent-detection flag is turned off, and the turned off event-detectionflag is output to the control unit 72.

A process for recording an image using an event-determination resultflag will be described with reference to FIGS. 23 and 24.

In an upper portion of FIG. 23, images G3′ displayed on the displaydevice 31′-3 (hereinafter also referred to as “screens G3′ of thedisplay device 31′-3”) at times T1 and T2 (time T1<time T2) areillustrated.

As shown in a lower portion of FIG. 23, at a time TO (time T0<time T1),a user A views a screen G2′ of the display device 31′-2 among screensG1′ to G9′ of the display devices 31′-1 to 31′-9. At the time T1, theuser A moves his/her line of sight to the screen G3′ of the displaydevice 31′-3. At the moment of the time T1 when the user A changeshis/her line of sight from the screen G2′ to the screen G3′, the user Anotices the presence of an object of interest (e.g., in the exampleshown in FIG. 23, as in the example shown in FIG. 6, the pitcher's gripbefore throwing the ball when the pitcher cocks his arm) on the screenG3′.

When the user A notices an object of interest to make a memo, in manycases, he/she holds the remote controller 112 and takes an action (e.g.,to move his/her finger) on the held remote controller 112. For example,to record the object on the screen G3′, at the time T1 when the user Anotices the object of interest on the screen G3′, the user A holds theremote controller 112 and presses the screen selection button 55-3 onthe front surface of the remote controller 112 to select the screen G3′of the display device 31′-3. Then, at the time T2, the user A pressesthe memo button 51. However, as described above with reference to FIG.6, the information on the screen G3′ has already changed at the momentof the time T2 when the user A presses the memo button 51, and the userA may fail to record the object displayed at the moment of the time T1on the screen G3′. Also in the example shown in FIG. 23, the screen G3′displayed at the time T2 shows an image of a moment when the pitcherthrows the ball.

Therefore, the acceleration sensor 121 of the remote controller 112detects a movement of the main body of the remote controller 112 or amovement of the user's finger, and records an image displayed on thescreen G3′ at the moment when such a movement is detected (i.e., thescreen G3′ displayed at the time T1) in the memory 62 in advance. Then,the image recorded in the memory 62 is used as the screen G3′ displayedat the time T2 when the user A actually presses the memo button 51.

This feature will be specifically described with reference to FIG. 24.In the example shown in FIG. 24, a change in the acceleration-sensordata for the remote controller 112 over time, and a change in the outputimage displayed on the screen G3′ over time are illustrated.

The display devices 31′-1 to 31′-9 monitor an incoming event-detectionflag from the remote controller 112. At the time T1 when a change inacceleration of the remote controller 112 occurs, the display devices31′-1 to 31′-9 store information displayed on the screens G1′ to G9′ inthe memories 62 as memory images #1 on the basis of the monitoredevent-detection flag. In this case, unlike a change in the line ofsight, which screen the user A is viewing is not determined by a changein acceleration of the remote controller 112. Therefore, imagesdisplayed on all the screens G1′ to G9′ at the time of the change inacceleration are stored as memory images #1 in the memories 62 of thedisplay devices 31′.

At the time T2 when the user A actually presses the memo button 51 forthe screen G3′, the display device 31′-3 stores the informationdisplayed at the time T2 on the screen G3′ in the memory 62 as a memoryimage #2. Further, as indicated by the output image on the screen G3′,the display device 31′-3 switches an output image on the screen G3′ fromthe image from the tuner 61 to the memory image #1. In the secondembodiment, as in the first embodiment, a presetting may be performed soas to switch an output image on the screen G3′ to the memory image #2instead of the memory image #1.

Accordingly, the moment when the user A moves the remote controller 112is estimated to be the moment when the user A desires to make a memo,and an image displayed when a rapid change in acceleration of the remotecontroller 112 occurs is recorded. Therefore, for example, even if theuser A hesitates to operate the remote controller 112, failure to catchan object of interest can be avoided.

A process of the remote controller 112 shown in FIG. 20 will bedescribed in detail with reference to a flowchart shown in FIG. 25. InFIG. 25, the processing of steps S154 to S156 is basically similar tothe processing of steps S51 to S53 shown in FIG. 11, respectively, and adetailed description thereof is thus omitted to avoid redundancy.

The acceleration sensor 121 constantly detects the acceleration of theremote controller 112, and outputs acceleration-sensor data indicatingthe detected acceleration to the AD conversion unit 122.

In step S151, the AD conversion unit 122 converts theacceleration-sensor data output from the acceleration sensor 121 fromanalog to digital (AD) form, and outputs the AD-convertedacceleration-sensor data to the event determination unit 123.

In step S152, the event determination unit 123 performs anevent-detection flag output process. The event-detection flag outputprocess will be described with reference to a flowchart shown in FIG.26.

In step S171, the event determination unit 123 determines whether or nota rapid change in acceleration has been detected on the basis of theacceleration-sensor data output from the AD conversion unit 122. If itis determined that a rapid change in acceleration has been detected, theprocess proceeds to step S172, and the event determination unit 123turns on the event-detection flag.

If it is determined in step S171 that no rapid change in accelerationhas been detected, the process proceeds to step S173, and the eventdetermination unit 123 turns off the event-detection flag.

In step S174, the event determination unit 123 outputs the turned on oroff event-detection flag to the control unit 72. Then, the processreturns to step S152 shown in FIG. 25, and proceeds to step S153.

Referring back to FIG. 25, in step S153, the control unit 72 determineswhether or not the event-detection flag has been turned on. If it isdetermined that the event-detection flag has been turned on, the processskips step S154 and proceeds to step S155. In this case, the controlunit 72 generates an event-determination result flag (which is turnedon) on the basis of the event-detection flag, and outputs the generatedevent-determination result flag to the frame generation unit 73. If itis determined in step S153 that the event-detection flag has not beenturned on, the process proceeds to step S154.

For example, when a user operates the operation unit 71 including, asshown in FIG. 3, the memo button 51, the zoom button 52, the Fn1 button53, the Fn2 button 54, the screen selection button 55, and the operatingstick 56, the control unit 72 generates a command recognizable by thedisplay devices 31′ on the basis of the operation signal input via theoperation unit 71 in accordance with the operation using the operationunit 71, and outputs the generated command to the frame generation unit73.

If the frame generation unit 73 determines in step S154 that a commandhas been input, the process proceeds to step S155. If it is determinedin step S154 that no command has been input, the process returns to stepS151, and the subsequent processing is repeated.

In step S155, the frame generation unit 73 generates a transmissionframe based on data such as the event-determination result flag orcommand from the control unit 72, and transmits the generatedtransmission frame to the transmission unit 74. In step S156, thetransmission unit 74 converts the transmission frame generated by theframe generation unit 73 into a light-emission signal to cause thelight-emitting unit 75 to emit light.

Thus, the light-receiving unit 32 of the display device 31′ receives thelight from the remote controller 112, and transmits data correspondingto the received light (i.e., the command or event-determination resultflag) to the display devices 31′.

A process for recording an image using an event-determination resultflag by the display device 31′ corresponding to the process shown inFIG. 25 will be described in detail with reference to a flowchart shownin FIG. 27. In FIG. 27, the processing of steps S201 and S204 to S210 isbasically similar to the processing of steps S11, S15 to S17, and S19 toS22 shown in FIG. 8, respectively, and a detailed description thereof isthus omitted to avoid redundancy.

In step S201, the control unit 66 checks a preset DIP switch or the liketo obtain the ID number of the display device 31′. In step S202, thecontrol unit 66 determines whether or not data has been received fromthe remote controller 112. If it is determined that data has beenreceived from the remote controller 112, the process proceeds to stepS203.

In step S203, the control unit 66 performs a memory control process. Thememory control process will be described with reference to a flowchartshown in FIG. 28.

In step S231, the control unit 66 determines whether or not anevent-determination result flag has been received. If the data receivedfrom the remote controller 112 in step S202 shown in FIG. 27 is anevent-determination result flag, it is determined in step S231 that anevent-determination result flag has been received, and the processproceeds to step S232. In step S232, the control unit 66 determineswhether or not the memory 62 is in a protected mode. If it is determinedthat the memory 62 is not in the protected mode, i.e., the protectedmode of the memory 62 has been released, the process proceeds to stepS233.

In step S233, the control unit 66 stores the image of the image signalinput to the memory 62 as a memory image #1. Specifically, the controlunit 66 updates the memory image #1 stored in the memory 62 with animage of an image signal input to the memory 62 at the time of thereception of the event-determination result flag (in other words, at thetime of the detected rapid change in acceleration) and output to thedisplay 65. Then, the process returns to step S203 shown in FIG. 27, andproceeds to step S204.

If it is determined in step S231 that no event-determination result flaghas been received, the process proceeds to step S234. That is, a commandhas been received. Then, in step S234, the control unit 66 resets thevalue of the internal counter. Then, the process returns to step S203shown in FIG. 27, and proceeds to step S204. If it is determined in stepS232 that the memory 62 is not in the protected mode, the process alsoreturns to step S203 shown in FIG. 27, and proceeds to step S204.

Referring back to FIG. 27, in step S204, the control unit 66 determineswhether or not the received command is a screen selection command. If itis determined in step S204 that the received command is a screenselection command, in step S205, the control unit 66 changes theselected mode of the screen of the display device 31 according to thescreen selection command. Then, the process returns to step S202, andthe subsequent processing is repeated.

If it is determined in step S204 that the received command is not ascreen selection command, in step S206, the control unit 66 determineswhether or not the screen of the display device 31 is in the selectedmode. If it is determined in step S206 that the screen is in theselected mode, in step S207, the control unit 66 performs a process inaccordance with the command. The process in accordance with the commandis similar to the process in accordance with the command described abovewith reference to FIG. 10, and a detailed description thereof is thusomitted to avoid redundancy.

That is, in step S207, if the received command is a memo command, theimage of the image signal input to the memory 62 is stored as a memoryimage #2; the memory 62 is set to the protected mode; and the selectionof the output selection unit 63 is controlled so as to read the imagesignal of the memory image #1 stored in the memory 62. Therefore, thedisplay 65 displays the memory image #1 recorded at the time of thedetected rapid change in acceleration.

If the received command is a memory-image selection command, the outputselection unit 63 is controlled so as to read the image signal of thememory image in accordance with the command (the memory image #1 or thememory image #2). Therefore, the display 65 displays the memory image #1recorded at the time of the detected rapid change in acceleration, ordisplays the memory image #2 recorded at the time of the reception ofthe memo command.

If the received command is a zoom command, zoom parameters (such as zoommagnification and zoom range) are transmitted, and the image processingunit 64 is controlled so as to perform image zoom processing on theimage signal. Therefore, the display 65 displays an image correspondingto the image signal subjected to the zoom processing on the basis of thezoom parameters.

After the processing of step S207, the process returns to step S202, andthe subsequent processing is repeated. If it is determined in step S206that the screen is not in the selected mode, the process also returns tostep S202, and the subsequent processing is repeated.

If it is determined in step S202 shown in FIG. 27 that no data has beenreceived from the remote controller 112, the process proceeds to stepS208. Since no data has been received from the remote controller 112, instep S208, the control unit 66 increments the value of the internalcounter. In the example shown in FIG. 27, therefore, the value of thecounter is not incremented even if an event-determination result flaghas been received, but is not reset because no command has beenreceived.

In step S209, the control unit 66 determines whether or not the value ofthe counter is equal to or more than a predetermined threshold value. Ifit is determined that the value of the counter is equal to or more thanthe predetermined threshold value, i.e., when a period of time not lessthan the threshold value has elapsed since the last command wasreceived, in step S210, the control unit 66 releases the protected modeof the memory 62. Therefore, a write to the memory 62 is permitted inaccordance with the event-determination result flag or the memo command.

If it is determined in step S209 that the value of the counter is lessthan the threshold value, the process skips step S210 and returns tostep S202, and the subsequent processing is repeated.

Accordingly, the moment when the acceleration of the remote controller112 rapidly changes is estimated to be the moment when the user desiresto make a memo, and the screens of all the display devices 31′ arerecorded in the memories 62 when a rapid change in acceleration of theremote controller 112 occurs. Therefore, for example, even if the userhesitates to operate the remote controller 112, failure to catch anobject of interest can be avoided.

Further, an image displayed on the display device 31′ as a target to beoperated (set to the selected mode) when a memo command is received isalso recorded in the memory 62 separately from the images recorded whena rapid change in acceleration occurs. Therefore, an image displayedwhen the user successfully operates the remote controller 112 can alsobe obtained.

Further, if there exists a memory image stored in the memory 62according to a memo command, the memory 62 is set to the protected modeso as to prevent the memory image from being overwritten. This canprevent the memory image stored in the memory 62 from being deleted by auser's accidental action such as moving his/her line of sight during theoperation of the remote controller 112.

In the foregoing description, an event-determination result flag isdelivered in response to a detection of the acceleration of the remotecontroller 112. Alternatively, a multi-display system may be configuredto deliver an event-determination result flag by capturing the event ofsurprise from speech uttered by a speaker and determining the event fromthe voice of the speaker.

FIG. 29 is a diagram showing another example of the electrical structureof the remote controller 112 shown in FIG. 20 (hereinafter referred toas a “remote controller 112 ′”). The remote controller 112′ shown inFIG. 29 includes a microphone 151 and an event determination unit 152 inplace of the acceleration sensor 121 and the event determination unit123 of the remote controller 112 shown in FIG. 22, respectively. As inremote controller 112 shown in FIG. 22, the remote controller 112′further includes an AD conversion unit 122, an operation unit 71, acontrol unit 72, a frame generation unit 73, a transmission unit 74, anda light-emitting unit 75.

The microphone 151 captures sound such as speech uttered by a speaker,and outputs an audio signal corresponding to the captured sound to theAD conversion unit 122. The AD conversion unit 122 converts the outputvalue (audio signal) output from the microphone 151 from analog todigital (AD) form, and outputs the AD-converted audio data to the eventdetermination unit 152.

As shown in FIG. 30, the event determination unit 152 shown in FIG. 29includes a buffer 161, a fast Fourier transform (FFT) unit 162, asound-pressure-level calculation unit 163, a buffer 164, a differencecalculation unit 165, and a determination unit 166.

The buffer 161 stores audio samples obtained from the audio data inputfrom the AD conversion unit 122. The FFT unit 162 applies an FFT to thedata stored in the buffer 161 under the control of the determinationunit 166 when the difference from the previous sample is large, andoutputs a result of the FFT to the determination unit 166.

The sound-pressure-level calculation unit 163 calculates a soundpressure level using the data of the several samples stored in thebuffer 161, and outputs data of the calculated sound pressure level tothe buffer 164 and the difference calculation unit 165. In the buffer164, data of the previous sound pressure level calculated by thesound-pressure-level calculation unit 163 is stored.

The difference calculation unit 165 calculates a difference between thesound pressure level output from the sound-pressure-level calculationunit 163 and the previous sound pressure level stored in the buffer 164,and outputs the difference to the determination unit 166.

The determination unit 166 determines the magnification of thedifference output from the difference calculation unit 165. If thedifference is large, the determination unit 166 controls the FFT unit162 to apply an FFT to the data stored in the buffer 161, and turns onor off the event-detection flag according to a result of the FFT.Specifically, as a result of the FFT, if a power concentration in alow-frequency part occurs, the event-detection flag is turned on. If nopower concentration in the low-frequency part occurs, theevent-detection flag is turned off. The turned on or off event-detectionflag is output to the control unit 72.

A process of the remote controller 112′ shown in FIG. 29 and the processof the remote controller 112 described above with reference to FIG. 25are basically similar to each other, except for the event-detection flagoutput process in step S152 (i.e., the process shown in FIG. 26). Thus,only the event-detection flag output process in step S152, which isdifferent from that shown FIG. 26, will be described with reference to aflowchart shown in FIG. 31.

In step S251, the buffer 161 obtains audio samples from the audio datainput from the AD conversion unit 122. In step S252, the audio samplesare stored in the buffer 161. In step S253, the sound-pressure-levelcalculation unit 163 calculates a sound pressure level using data of theseveral samples stored in the buffer 161, and outputs the data of thecalculated sound pressure level to the buffer 164 and the differencecalculation unit 165.

In step S254, the difference calculation unit 165 calculates adifference between the sound pressure level output from thesound-pressure-level calculation unit 163 and the previous soundpressure level stored in the buffer 164, and outputs the difference tothe determination unit 166. In step S255, the determination unit 166determines whether or not the difference output from the differencecalculation unit 165 is large. If it is determined that the differenceoutput from the difference calculation unit 165 is large, the processproceeds to step S256, and the determination unit 166 controls the FFTunit 162 to apply an FFT to the data stored in the buffer 161.

In step S257, the determination unit 166 determines whether or not apower concentration in the low-frequency part occurs as a result of theFFT by the FFT unit 162. If it is determined that a power concentrationin the low-frequency part occurs, in step S258, the determination unit166 turns on the event-detection flag.

If it is determined in step S255 that the difference output from thedifference calculation unit 165 is not large, or if it is determined instep S257 that no power concentration in the low-frequency part occurs,the process proceeds to step S259, and the determination unit 166 turnsoff the event-detection flag.

In step S260, the determination unit 166 outputs the turned on or offevent-detection flag to the control unit 72. Then, the process returnsto step S152 shown in FIG. 25.

Accordingly, the remote controller 112′ is provided with the microphone151, and is configured such that the moment when a user utters anexclamation of surprise is estimated to be the moment when the userdesires to make a memo and all screens are recorded in the memories 62when a power concentration in the low-frequency part of the voice of theuser occurs. Therefore, for example, even if the user hesitates tooperate the remote controller 112′, failure to catch an object ofinterest can be avoided.

In the foregoing description, the occurrence of a power concentration inthe low-frequency part of the voice of the user is detected as theutterance of an exclamation of surprise from the user. Alternatively,voice with a certain volume level or a particular type of voice can bedetected.

Accordingly, the moment when the user changes his/her line of sight, themoment when the remote controller 112′ is moved, or the moment when theuser utters speech is estimated to be the moment when the user desiresto make a memo, and screens displayed when any of those events isdetected are recorded in the memories 62. Therefore, for example, evenif the user hesitates to operate the remote controller 112′, failure tocatch an object of interest can be avoided.

Further, no hard disk is used, resulting in no increase in cost.

While the recording of a moving image has been described above withreference to FIGS. 13 to 19 in the context of the multi-display systemshown in FIG. 2, a moving image can be recorded using the multi-displaysystem shown in FIG. 20. Also in the multi-display system shown in FIG.20, a rapid change in acceleration or a power concentration in thelow-frequency part of voice can be used as a trigger to record not onlya still image but also a moving image.

Specifically, in the multi-display system shown in FIG. 2 or 20, themoment when the user changes his/her line of sight, or the moment whenthe remote controller 22 or 112 is moved, or a period of time after theuser has uttered speech is estimated to be a moment or a period of timeat or in which information of which the user desires to make a memo isshown, and a moving image displayed when such an event was detected isrecorded in the memory 62. Therefore, for example, even if the userhesitates to operate a remote controller 22 or 112, failure to catch anobject of interest can further be avoided.

In the foregoing description, any information such as line-of-sightdetection data or a change in acceleration or speech detected in theremote controller 22 or 112 is used to estimate a moment when the userfinds an object of interest, and an image is recorded, whereby failureto catch the object can be avoided. Alternatively, both line-of-sightdetection data and a change in acceleration, or both line-of-sightdetection data and a change in speech, or both a change in accelerationand a change in speech may be used. Alternatively, all those three typesof information may be used.

In the foregoing description, images of different programs are displayedon display devices. In an embodiment of the present invention, as far asa plurality of images can be displayed on a multi-display apparatus, asingle enlarged image may be displayed on at least two of the displaydevices, and images of different programs may be displayed on the otherdisplay devices.

The series of processes described above can be implemented by hardwareor software.

When the series of processes are implemented by software, programsconstituting the software is installed from a program recording mediumonto a computer incorporated in special hardware, or a general-purposepersonal computer capable of executing various functions with variousprograms installed thereon, or the like.

FIG. 32 is a block diagram showing an example of the structure of apersonal computer 301 that executes the series of processes describedabove according to a program. A central processing unit (CPU) 311executes various processes according to programs stored in a read-onlymemory (ROM) 312 or a storage unit 318. A random access memory (RAM) 313stores programs executed by the CPU 311, data, etc., as necessary. TheCPU 311, the ROM 312, and the RAM 313 are connected to each other via abus 314.

The CPU 311 is also connected to an input/output interface 315 via thebus 314. The input/output interface 315 is connected to an input unit316 including the sensor 121 described above, a keyboard, a mouse, amicrophone, etc., and to an output unit display 317 including a display,a loudspeaker, etc. The CPU 311 executes various processes according toinstructions input from the input unit 316. The CPU 311 then outputsresults of the processes to the output unit 317.

The storage unit 318 connected to the input/output interface 315 isformed of, for example, a hard disk, and stores programs executed by theCPU 311 and various data. A communication unit 319 communicates with anexternal apparatus via a network such as the Internet or a local areanetwork.

Also, the programs may be obtained via the communication unit 319 andstored in the storage unit 318.

A drive 320 is further connected to the input/output interface 315. Whena removable medium 321 such as a magnetic disk, an optical disk, amagneto-optical disk, or a semiconductor memory is mounted onto thedrive 320, the drive 320 drives the removable medium 321 to obtainprograms, data, etc., recorded thereon. The obtained programs, dataetc., are transferred to and stored in the storage unit 318, asnecessary.

As shown in FIG. 32, a program recording medium storing programs that isinstalled onto a computer so as to be executable by the computer may beformed of the removable medium 321, which is a package medium such as amagnetic disk (including a flexible disk), an optical disk (including acompact disc read-only memory (CD-ROM) and a digital versatile disc(DVD)), a magneto-optical disk, or a semiconductor memory, the ROM 312in which programs are temporarily or persistently stored, a hard diskconstituting the storage unit 318, or the like. The programs can bestored in the program recording medium via the communication unit 319,which is an interface such as a router or a modem, as necessary, usingwired or wireless communication medium such as a local area network, theInternet, or digital satellite broadcasting.

In this specification, steps defining the programs stored in the programrecording medium may include processes that are executed in parallel orindividually, as well as processes that are executed in the ordersdescribed herein.

In this specification, a system refers to the entirety of a plurality ofdevices.

Embodiments of the present invention are not limited to those describedabove, and a variety of modifications can be made without departing fromthe scope of the present invention.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

1. A display control system for controlling display of images on aplurality of display means, the plurality of display means beingarranged in an array, the display control system comprising: determiningmeans for determining whether or not information of interest is beingdisplayed; first recording control means for controlling recording of afirst image displayed on the display means when the determining meansdetermines that the information of interest is being displayed; secondrecording control means for controlling recording of a second imagedisplayed on the display means when a user performs a recordingoperation; and display control means for controlling display of thefirst image or the second image on the display means.
 2. The displaycontrol system according to claim 1, further comprising a plurality ofdisplay control apparatuses each associated with each of the pluralityof display means and controlling display of an image on the displaymeans, each of the plurality of display control apparatuses including:the determining means; the first recording control means; the secondrecording control means; and the display control means.
 3. The displaycontrol system according to claim 1, further comprising the plurality ofdisplay means.
 4. The display control system according to claim 2,further comprising a line-of-sight detection apparatus that detects aline of sight of the user, wherein the determining means of each of theplurality of display control apparatuses determines whether or not theinformation of interest is being displayed on the display means on thebasis of a detection result from the line-of-sight detection apparatus.5. The display control system according to claim 4, wherein the firstrecording control means of each of the plurality of display controlapparatuses controls the recording of the first image displayed on thedisplay means when the determining means determines that the informationof interest is being displayed on the display means on the basis of thedetection result from the line-of-sight detection apparatus.
 6. Thedisplay control system according to claim 2, further comprising a remotecontroller that controls the plurality of display control apparatuses,wherein the remote controller includes speech detecting means fordetecting specific speech of the user, and the determining means of eachof the plurality of display control apparatuses determines whether ornot the information of interest is being displayed on one of theplurality of display means on the basis of a detection result from thespeech detecting means.
 7. The display control system according to claim6, wherein the first recording control means of each of the plurality ofdisplay control apparatuses controls the recording of the first imagedisplayed on the display means when the determining means determinesthat the information of interest is being displayed on one of theplurality of display means on the basis of the detection result from thespeech detecting means.
 8. The display control system according to claim2, further comprising a remote controller that controls the plurality ofdisplay control apparatuses, wherein the remote controller includesacceleration detecting means for detecting an acceleration of the remotecontroller that is operated by the user, and the determining means ofeach of the plurality of display control apparatuses determines whetheror not the information of interest is being displayed on one of theplurality of display means on the basis of a detection result from theacceleration detecting means.
 9. The display control system according toclaim 8, wherein the first recording control means of each of theplurality of display control apparatuses controls the recording of thefirst image displayed on the display means when the determining meansdetermines that the information of interest is being displayed on one ofthe plurality of display means on the basis of the detection result fromthe acceleration detecting means.
 10. The display control systemaccording to claim 2, wherein the first recording control means of eachof the plurality of display control apparatuses controls to startrecording of a moving image, starting from the first image displayed onthe display means, when the determining means determines that theinformation of interest is being displayed, the second recording controlmeans controls to stop the recording of the moving image at the secondimage displayed on the display means when the user performs therecording operation, and the display control means controls display ofthe moving image starting with the first image displayed on the displaymeans and ending with the second image displayed on the display means.11. A display control method for a display control system forcontrolling display of images on a plurality of display means, theplurality of display means being arranged in an array, the displaycontrol method comprising the steps of: determining whether or notinformation of interest is being displayed; controlling recording of afirst image displayed on the display means when it is determined thatthe information of interest is being displayed; controlling recording ofa second image displayed on the display means when a user performs arecording operation; and controlling display of the first image or thesecond image on the display means.
 12. A display control apparatus in adisplay control system including a plurality of display controlapparatuses, for controlling display of an image on display meansassociated with the display control apparatus from among a plurality ofdisplay means arranged in an array, the display control apparatuscomprising: the display means for displaying an image corresponding toan image signal input to the display control apparatus; determiningmeans for determining whether or not information of interest is beingdisplayed; first recording control means for controlling recording of afirst image displayed on the display means when the determining meansdetermines that the information of interest being displayed; secondrecording control means for controlling recording of a second imagedisplayed on the display means when a user performs a recordingoperation; and display control means for controlling display of thefirst image or the second image on the display means.
 13. A displaycontrol method for a display control apparatus in a display controlsystem including a plurality of display control apparatuses, forcontrolling display of an image on display means associated with thedisplay control apparatus from among a plurality of display meansarranged in an array, the display control method comprising the stepsof: determining whether or not information of interest is beingdisplayed; controlling recording of a first image displayed on thedisplay means associated with the display control apparatus when it isdetermined that the information of interest is being displayed;controlling recording of a second image displayed on the display meansassociated with the display control apparatus when a user performs arecording operation; and controlling display of the first image or thesecond image on the display means associated with the display controlapparatus.
 14. A program for allowing a display control apparatus in adisplay control system including a plurality of display controlapparatuses to execute a process for controlling display of an image ondisplay means associated with the display control apparatus from among aplurality of display means arranged in an array, the program comprisingthe steps of: determining whether or not information of interest isbeing displayed; controlling recording of a first image displayed on thedisplay means associated with the display control apparatus when it isdetermined that the information of interest is being displayed;controlling recording of a second image displayed on the display meansassociated with the display control apparatus when a user performs arecording operation; and controlling display of the first image or thesecond image on the display means associated with the display controlapparatus.
 15. A display control system for controlling display ofimages on a plurality of display units, the plurality of display unitsbeing arranged in an array, the display control system comprising: adetermining unit determining whether or not information of interest isbeing displayed; a first recording control unit controlling recording ofa first image displayed on the display units when the determining unitdetermines that the information of interest is being displayed; a secondrecording control unit controlling recording of a second image displayedon the display units when a user performs a recording operation; and adisplay control unit controlling display of the first image or thesecond image on the display units.
 16. A display control apparatus in adisplay control system including a plurality of display controlapparatuses, for controlling display of an image on a display unitassociated with the display control apparatus from among a plurality ofdisplay units arranged in an array, the display control apparatuscomprising: the display unit for displaying an image corresponding to animage signal input to the display control apparatus; a determining unitdetermining whether or not information of interest is being displayed; afirst recording control unit controlling recording of a first imagedisplayed on the display unit when the determining unit determines thatthe information of interest being displayed; a second recording controlunit controlling recording of a second image displayed on the displayunit when a user performs a recording operation; and a display controlunit controlling display of the first image or the second image on thedisplay unit.